Explores architectural changes aimed at improving capacity per parameter, even if slower than typical leaderboard approaches:

• ASQU (Asymmetric Squared Unit): learned per-channel generalization of ReLU^2
• MoC (Mixture of Convolutions): token-conditioned mixture of short convolutions (dynamic per-token kernels)
• BankedLinear: shared weight bank across layers (small learned + large fixed random set) with learned per-layer mixing

Ablations use the base train_gpt.py script for a fixed 10k steps. MLP expansion is adjusted to match size.

• ASQU replaces ReLU^2
• Short conv / MoC is applied to QKV
• BankedLinear replaces QKV projections

Results

Results are currently single-seed (1337); additional runs in progress.

ASQU: per-channel parameterization gives ~0.001 bpb improvement over scalar, where the scalar form converges to behavior similar to leaky ReLU^2 with slope 0.5

Also explored learning the exponent instead of fixing the square. While it did not consistently improve final performance (and was more expensive), it revealed a consistent depth-dependent pattern:

• early layers ~1.4
• middle layers ~1.8
• late layers ~2.2

With MoC, experiments of generating the dynamic kernel via learned projection performed poorly, suggesting the use of a more constrained mechanism (e.g. basis interpolation) is necessary for stable optimization.

For BankedLinear, one scalar weight is used per bank entry to construct the mixture. Experiments with per head weights worsened performance.

Experiments were done with a layer specific LoRA, but this worsened performance compared to just investing capacity in the MLP.